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DEVICES AND METHODS FOR IMAGING WITH CONTINUOUS TILT 

MICROMIRROR ARRAYS 



S Introduction 

This patent application claims the benefit of priority 
from U.S. provisional patent application Serial No. 
60/516,360, filed October 31, 2003, which is herein 
incorporated by reference in its entirety. 
10 This invention was supported in part by funds from the 

U.S. government (NSF Grant No. DMS-0211283) . The U.S. 
government may therefore have certain rights in the 
invention. 

IS Field of the Invention 

The present invention relates to devices and methods of 
use of devices comprising a photographic imaging system, a 
micromirror array and a pixel assembly system or algorithm. 
The devices of the present invention provide for real time, 

20 high resolution image acquisition by mosaicing of pixels of 
a scene reflected by micromirrors of the micromirror array 
when mirrors are tilted individually in at least two 
different directions. The pixels are collected by the 
photographic imaging system and reassembled by the pixel 

25 assembly system or algorithm into a high resolution image of 
the scene. 

Background of the Invention 

For decades, the lens -CCD chip paradigm has dominated 
30 visual sensor design. An important design goal of visual 
sensors is to enhance or increase their resolution. 

To create a very high resolution image, typically 
several images are taken and then pasted together. This 
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method, known as mosaicing can yield impressive results. 
However, with conventional devices the images are obtained 
slowly, since the camera must be moved many times. Further, 
the introduction of mechanical devices into imaging systems 
5 has largely been avoided, because moving a macroscopic 
camera at the required speeds is difficult and is 
potentially damaging to the camera. 

A system for creating spherical mosaics using a zoom 
lens and a pan- tilt mechanism mounted on a robot was 

10 described by Kropp et al . (Proceeding IEEE Workshop on 
OmniDirectional Vision 2000 47-53) . However, the 

acquisition process is slow since the camera position must 
be moved many times to capture a complete image. 

Another approach has been to fix the camera's position 

15 and place a mirror in front of the camera, which can then be 
moved to create a mosaic with increased field of view 
(Nakao, T. and Kashitani, A. International Conference on 
Image Processing, Oct 7-10, 2001, 2:1045-1048). However, 
manual movement of the mirror also has disadvantages and is 

20 still slow. 

During the past twenty years, the field of micro- 
electro-mechanical systems (MEMS) has developed remarkable 
capabilities and the possible applications are only 
beginning to be grasped. 

25 One micro-optical-electro-mechanical system (MOEMS) is 

the digital micromirror device or DMD. The DMD was 
developed at Texas Instruments over a period of years, 
starting in the 1970s. This device consists of a chip 
covered with an array of small mirrors, each of whose 

30 orientation may be separately controlled. The typical size 
of an individual micromirror is approximately 15 pm square 
and is made of a highly reflective aluminum alloy. One of 
the main advantages of being small for an optical device is 
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the ability to change state rapidly. In the case of a DMD, 
this can be in the megahertz range. 

The primary use of MOEMS has been in projectors. In 
projection, one or more micromirrors corresponds to a single 
5 pixel in a projected image. Different light is reflected by 
the micromirrors and the relative amount of time each mirror 
is in the "on" or "off" position when red, blue or green 
light shines on it determines the hue of and shade of the 
pixel it generates. A projector incorporating micromirrors 

10 operates by reflecting light rays from an external source 
into the pupil of an imaging lens. The imaging lens then 
projects the digitized image onto the screen. 

Micromirror arrays are also used in microscopy, retinal 
scanning and lithography. 

15 In these applications, the micromirror array generally 

acts as a mask, and the individual mirror elements have only 
two states (Hornbeck, L. Texas Instruments Technical Journal 
1998 15 :7-46) . 

Micromirror arrays are also used in optical switching 
20 (Wu, M.C. Proceedings of the IEEE 1997 85 (11) : 1833-1856) . 
Optical switching is an application where continuous pitch 
micromirror arrays have been used. For example, U.S. Patent 
6,600,651 discloses an optical switch comprising mirror 
elements each of which can be assigned an arbitrary 
25 orientation, i.e. each mirror has a full 2 degrees of 
freedom with respect to tilt. A fiber optic communication 
system utilizing MEMS tilting mirrors is also disclosed in 
U.S. Patent 6,690,885. 

U.S. Patent 6,700,606 discloses an optical imager with 
30 a light source, a platen for reflecting a portion of light 
emitted by the light source, an image sensor for sensing the 
light and a micromirror device with a first position which 
reflects light reflected from a first location on the platen 
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and a second position which reflects light reflected from a 
second location on the platen to the image sensor. This 
optical imager is suggested to be useful in imaging a 
fingerprint placed against the platen. 
5 Nayar et al . describe a programmable imaging system 

comprising a digital binary micromirror array, which 
provides an image by taking a single picture of the array 
(XCVPR 2004 1:436-443). In this system the micromirror 
array acts as a mask. 

10 

Summary of the Invention 

An object of the present invention is to provide a 
device for forming high resolution images which comprises a 
photographic imaging system and a micromirror array 

15 containing an array of micromirrors, each mirror being 
capable of tilting in at least two directions. The 
micromirror array is positioned with respect to the 
photographic imaging system so that each mirror of the 
micromirror array transfers a reflected pixel of the scene 

20 to be photographed to the photographic imaging system. The 
device further comprises an assembly system or algorithm 
which forms a high resolution image of the scene by 
mosaicing relevant color values extracted from each 
reflected pixel from each mirror of the micromirror array 

25 into a high resolution image of the scene. 

Another object of the present invention is to provide a 
method for producing a high resolution image of a scene via 
a photographic imaging system by incorporating into the 
photographic imaging system a micromirror array positioned 

30 with respect to the photographic imaging system to be 
capable of transferring reflected pixels of the scene to the 
photographic imaging systems. In the method of the present 
invention, a high resolution image of the scene is 
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reassembled algorithmically by mosaicing relevant color 
values extracted from the reflected pixels from the mirrors 
of the micromirror array when each mirror is tilted 
individually in at least two different directions. 

5 

Brief Description of the Figures 

Figure 1A and IB show schematic diagrams of the basic 
concept of image acquisition used in the device of the 
present invention. Figure 1A provides a diagram wherein the 

10 mirrors of the micromirror array are tilted to reflect 
pixels of points 1, 3, 5, 7 and 9 of the scene to the 
photographic imaging system, in this example a camera. In 
Figure IB, the state of the mirrors has been changed to 
transfer a reflection of the pixels at points 2, 4, 6, 8 and 

15 10 of the scene. 

Figure 2A and 2B show results from a simulation 
experiment of a photograph of a desktop. Figure 2A is a 
photograph of an overhead view of a desktop scene taken with 
a simulated pinhole camera. Figure 2B is a 640 by 448 image 

20 of the same scene created by mosaicing color values 
corresponding to pixels reflected from a simulated 
micromirror array in accordance with the present invention. 
To produce this image of Figure 2B, 70 images of a 64x64 
mirror array in different configurations were generated, 

25 subsampled and joined or mosaiced together. 

Detailed Description of the Invention 

In the present invention, an optical MEM or MOEM system 
comprising a micromirror array consisting of a plurality of 
30 microscopic mirrors capable of tilting individually at least 
two different directions and changing state thousands to 
millions of times per second, is incorporated into a 
photographic imaging system with an assembly system or 
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mosaicing assembly algorithm to provide devices and methods 
for use of such devices to produce high resolution 
photographic images . 

In the devices of the present invention, a micromirror 
5 array is positioned with respect to a photographic imaging 
system so that each mirror of the micromirror array 
transfers a reflected pixel of the scene to be photographed 
to the photographic imaging system. The device further 
comprises an assembly system or algorithm which forms a high 

10 resolution image of the scene by mosaicing each reflected 
pixel as a color value from each mirror of the micromirror 
array into a high resolution image of the scene. In one 
embodiment, the photographic imaging system is linked to a 
separate computer with the assembly algorithm. 

15 One configuration for the photographic imaging system 

and micromirror array of the device of the present invention 
is depicted in Figure 1A and IB. As shown therein, a 
photographic imaging system 2 such as a video camera or 
other camera capable of collecting digital images is pointed 

20 at a micromirror array 3 . The micromirror array 3 is 
positioned with respect to the photographic imaging system 2 
and the scene 4 to be imaged so that a reflection of the 
scene 4 is transferred to the photographic imaging system 2 . 
Thus, the micromirror array acts in similar fashion to a 

25 conventional mirror with the exception that it can 
reconfigure the state of the mirrors very rapidly. 
Preferably, each mirror of the array is individually 
controllable, and has at least 2 tilt directions so that 
instead of moving the photographic imaging system to obtain 

30 multiple images of a scene, the mirrors of the array move to 
produce different reflections of the same scene. In order 
to form a high resolution image of the scene using this 
device, one photographs the array with the mirrors in many 
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different states. The final resulting image of the scene is 
assembled via an assembly system or algorithm from reflected 
pixels and corresponding color values extracted from the 
♦reflected pixels from each mirror. Thus, for each high 
5 resolution image or photograph the individual mirrors are 
imaged, and each is individually oriented so that from the 
photographic imaging system's point of view it reflects a 
small portion of the scene. From each reflected pixel from 
each mirror one or more color values may be extracted. 

10 These correspond to points in the scene. Since the position 
of the micromirror array and relative position of the 
photographic imaging system are known, it is possible to 
then place or assemble the color values into a matrix that 
forms a final photograph of the scene. In other words, the 

15 positions of the pixel in the final image are determined by 
the basic known geometric quantities of the device and the 
elements thereof . The greater the number of states of the 
mirrors, the larger the number of color values extracted, 
and so the higher the resolution of the resulting image. In 

20 this device, the individual images are recorded at frame 
rate, i.e. the rate at which images are viewed on television 
(or even faster) . All of the images for the resulting 
photograph of the scene can thus be obtained very rapidly, 
and then assembled into a single highly resolved image via 

25 an assembly system or algorithm. Such algorithms are well 
known to those skilled in the art and can be programmed in 
multiple computer languages including, but in no way limited 
to Mat lab TM or C. The algorithms are based upon the 
geometry of the device of the present invention and the 

30 elements thereof. 

Examples of photographic imaging systems into which the 
MOEM or MEM system can be incorporated include but are not 
limited to digital cameras and video cameras. 
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An exemplary MOEM system useful in the present 
invention is the Lucent LambdaRouter array. This MOEM 
system is described in detail in U.S. Patent 6,690,885, the 
teachings of which are herein incorporated by reference in 
5 their entirety. The Lucent LambdaRouter array is a 16x16 
array with each mirror having a tilt angle of + 8 degrees. 
It has been estimated that each of the two angles of this 
array is controllable to within 0.05 degrees thus providing 
for 320 states in each angle (Gasparyan et al . In post 

10 deadline paper PD36-1, OFC, 20 03) . However, as will be 
understood by those skilled in the art upon reading this 
disclosure, any micromirror array with mirrors which can be 
tilted individually in at least two directions can be used. 
Micromirror arrays for use in imaging systems in 

15 accordance with the present invention are preferably 
constructed on surfaces using silicon substrates in 
accordance with established technologies for MEMS and MOEMS . 

Assembly systems or algorithms capable of mosaicing the 
reflected pixels of each mirror of the micromirror arrays 

20 into a single image are well known to those skilled in the 
art . These algorithms are based upon the geometry of the 
device of the present invention and the elements thereof. 
Such algorithms can be programmed in multiple computer 
languages including, but in no way limited to Mat lab TM or 

25 C. 

The resolution of an image acquired using the device of 
the present invention is only limited by the number of 
mirrors that are observable times the number of distinct 
states achievable by an individual mirror. Thus, the 
30 resolution achievable for a device of the present invention 
comprising, for example, a Lucent LambdaRouter array (with 
an estimated 320 states in each angle and a total number of 
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distinct states in an individual being on the order of 
100,000) is on the order of 25 megapixels. 

Thus the devices of the present invention provide a 
means for greatly increasing the current resolution of 
5 photographic imaging systems such as digital cameras and 
video cameras. These devices can also greatly enhance the 
resolution of omnidirectional surveillance cameras which 
often suffer from low resolution. 

The following nonlimiting example is provided to 
10 further illustrate the present invention. 

EXAMPLE 

The ability of a device of the present invention to 
increase image resolution was demonstrated using a geometric 
raytracing simulation. 

15 In this simulation, a 64x64 micromirror array having 2 

degrees of freedom was assumed. Each individual mirror was 
assumed to be configurable in any orientation so that the 
simulated device could exactly realize the distributions 
needed to perform mosaicing. Seventy ray tracing 

20 simulations were performed with this array to create a 

640x448 image of a desktop scene. The resolution of each of 
the 70 images was 640x480 and had to be subsampled to remove 
the rgb values from each of the mirrors . 

To create each of the 70 images, the simulated array 

25 was placed above the table and tilted down at 45 degrees. 
The 70 different distributions were calculated, each one 
corresponding to a small tile in the image. Using this 
information, the orientation of the mirrors was calculated 
and an image generated by the raytracer. 

30 To obtain higher resolution images, more than one 

sample rgb value could be extracted from each micromirror, 
an array with more micromirrors could be used, or more 
images could be acquired. 



